Skip to content

refactor: extract dereference/validate pipeline from reconcile#824

Merged
maltesander merged 18 commits into
mainfrom
feat/dereference-validate-steps
Jun 2, 2026
Merged

refactor: extract dereference/validate pipeline from reconcile#824
maltesander merged 18 commits into
mainfrom
feat/dereference-validate-steps

Conversation

@maltesander
Copy link
Copy Markdown
Member

Description

  • added dereference and validate steps
  • added test snapshots for smoke test.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

maltesander and others added 14 commits May 21, 2026 15:25
Frees up the `controller/` directory for upcoming dereference and validate
submodules, matching the trino-operator layout.

No behavior change.
Move all pre-loop client I/O (ZK ConfigMap, OPA URL, S3 connection, deep
storage bucket, AuthenticationClasses) into a new
`controller::dereference` submodule that returns `DereferencedObjects`.

Matches the trino-operator pattern (trino-operator commit 7004062).
No behavior change.
Move pure validation (product image resolution, TLS security construction,
authentication config conversion, product-config validation) into a new
`controller::validate` submodule that returns `ValidatedInputs`.

Matches the trino-operator pattern (trino-operator commit 7004062).
No behavior change.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Follow-up to the druid_controller → controller module rename.
Move the StatefulSet and PodDisruptionBudget shape assertions into
the upcoming 52-assert (resource shape) so that 50-assert mirrors
trino-operator's 10-assert (readiness-only). The duplicated broker
PDB and missing coordinator PDB will be corrected in 52-assert.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Declarative shape assertions for every operator-managed druid
resource in the smoke test (5 StatefulSets, 3 ClusterIP + 5 headless
Services, 3 Listeners, 5 PDBs, the shared internal Secret, the
ServiceAccount, and the RoleBinding). Mirrors trino's 13-assert
and kafka's 33-assert.yaml.j2 patterns. The duplicative *-metrics
Services are intentionally not asserted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fill in the bare log-config/log volume entries on the historical
StatefulSet so the assertion matches the live shape, and add the
selector blocks on the five headless Services (mirrors the
trino/kafka equivalents). Also clarify the vector_enabled comment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First slice of the ConfigMap snapshot assertion: the discovery
ConfigMap and the operator-managed Secret's key presence. The five
role-group ConfigMap snapshots follow in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds full .data snapshots for druid-{broker,coordinator,historical,
middlemanager,router}-default ConfigMaps. The actual side is
normalized for namespace and the random znode UUID before diffing.
The current snapshot was captured with VECTOR_AGGREGATOR unset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vector_enabled was declared in 52-assert.yaml.j2 as forward
scaffolding but has no consumer. Drop it; it can be re-introduced
in the commit that actually uses it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous check listed druid-router-default-0 twice and never
checked druid-broker-default-0. Replace the duplicate with the
broker entry and sort the list by role name for readability.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When VECTOR_AGGREGATOR is set, the operator adds a vector.yaml key
to each role-group ConfigMap; the other four keys are unchanged.
The vector.yaml payload is the same across all five roles except
for a single `.role = "<role>"` line, so the content is declared
once via a jinja2 `{% set %}` block and substituted per role inside
the heredoc.

Verified against a live VECTOR_AGGREGATOR=vector-aggregator-discovery
cluster in namespace kuttl-test-distinct-camel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `{% if vector_enabled -%}` and `{%- endif %}` blocks had
6 leading spaces. When vector_enabled was false, jinja2's strip
markers joined the leading whitespace of both blocks, putting
YAMLEOF at 12 spaces in the rendered output. After the kuttl
YAML block scalar strips 6 common-leading spaces, YAMLEOF ended
up at 6 spaces inside the script body, but the heredoc opener
is `<<'YAMLEOF'` (not `<<-`), which requires the terminator at
column 0. The heredoc therefore never closed and `\$(...)` never
terminated, producing `Syntax error: end of file unexpected
(expecting ")")` from dash.

Moving `{% if %}` and `{% endif %}` to column 0 (no strip markers)
mirrors kafka-operator's 34-assert.yaml.j2 pattern and renders
both branches with YAMLEOF at the correct indent.

Verified:
- non-vector render: 8 scripts pass against
  kuttl-test-fluent-mastodon
- vector render heredoc parses to the 5 expected keys
  (jvm.config, log4j2.properties, runtime.properties,
  security.properties, vector.yaml)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@adwk67 adwk67 self-requested a review June 1, 2026 09:42
@adwk67 adwk67 moved this to Development: In Review in Stackable Engineering Jun 1, 2026
Comment thread tests/templates/kuttl/smoke/52-assert.yaml.j2
Comment thread rust/operator-binary/src/controller.rs Outdated
Copy link
Copy Markdown
Member

@adwk67 adwk67 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - ran nightly test suite locally.

@maltesander maltesander enabled auto-merge June 2, 2026 12:54
@maltesander maltesander added this pull request to the merge queue Jun 2, 2026
@adwk67 adwk67 moved this from Development: In Review to Development: Done in Stackable Engineering Jun 2, 2026
Merged via the queue into main with commit d7d2e08 Jun 2, 2026
12 checks passed
@maltesander maltesander deleted the feat/dereference-validate-steps branch June 2, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: Done

Development

Successfully merging this pull request may close these issues.

2 participants